fix(otel): scope deterministic ID generation by zhongkechen · Pull Request #627 · aws/aws-durable-execution-sdk-java

zhongkechen · 2026-08-15T18:30:05Z

Summary

scope deterministic trace and span ID overrides to durable-plugin span creation while preserving the configured provider ID generator for unrelated instrumentation
consume scoped trace IDs before synchronous samplers or span processors run, preventing reentrant instrumentation from inheriting durable IDs
separate the deterministic Workflow trace from ambient Invocation traces and remove fabricated replay/continuation links
late-bind the ADOT/global tracer provider at invocation start; disable telemetry for the entire invocation when it is unavailable and retry on the next invocation without installing OpenTelemetry's no-op global
keep explicit builder-based providers eagerly bound and caller-owned
remove the unreleased plugin-owned AUTO_OTLP pipeline and redundant ProviderSource; config/no-arg constructors use the global provider and builder constructors use caller-owned providers
remove exporter and semantic-conventions dependencies that were only needed by AUTO_OTLP
remove OpenTelemetry handlers, tests, dependencies, and ADOT deployment plumbing from the examples module
rely on the Java-owned OpenTelemetry conformance workflow and handlers merged in feat(otel): Add per-PR OTel conformance test suite #630; the duplicate reusable-workflow caller previously added by this PR has been dropped

Testing

mvn spotless:check
OTel reactor tests: 161 passed
examples reactor tests: 103 total, with 72 passed and 31 cloud-only tests skipped
packaged conformance-tests-otel against this PR's installed SDK and OTel plugin artifacts
parsed all GitHub workflow YAML files locally
resolved examples dependency tree contains no OTel plugin, OpenTelemetry, or X-Ray artifacts

General cloud tests were not run locally because they require deployed AWS resources. OpenTelemetry end-to-end behavior is covered by the conformance workflow merged in #630.

Closes #625

Sign in to view

+    }
+
+    String generateTraceIdForExecution(String arn, Instant executionStartTime) {
+        var timestamp = executionStartTime != null ? executionStartTime : Instant.now();


github-actions · 2026-08-18T22:57:19Z

+        var timestamp = executionStartTime != null ? executionStartTime : Instant.now();
+        var timestampHex = String.format("%08x", timestamp.getEpochSecond() & 0xffffffffL);
+        return timestampHex + sha256(arn != null ? arn : "").substring(0, 24);


Claude AI review

The Workflow trace ID's 8-hex timestamp prefix is derived from executionStartTime, and the Workflow span is exported only on the terminal invocation. AWS X-Ray rejects trace IDs whose epoch prefix falls outside its ingestion window (traces older than ~30 days, or in the future). Because durable executions can run far longer than that window (the SDK supports up to a year), a long-running execution will export its Workflow root with a start timestamp that is weeks or months in the past, so X-Ray silently drops it — the deterministic Workflow trace that is the entire point of this feature disappears for exactly the long-running case, while short executions look fine. The conformance suite only launches delays up to 86400s (~23h), so this gap is not caught in CI.

There is a genuine tension here: for the Workflow root to be a single logical trace across every invocation, the trace ID must be byte-identical each time, so the timestamp cannot simply be "now" — no timestamp is both stable across a 1-year execution and always inside X-Ray's window. The team should decide explicitly which property to sacrifice and document it: e.g. accept and document that Workflow-trace correlation in X-Ray only holds within the ingestion window, and/or fall back to a non-X-Ray-timestamped scheme when the execution age exceeds the window. At minimum, add a long-execution case (or a documented known-limitation note) so this failure mode is not silent.

github-actions · 2026-08-18T22:57:21Z

Claude AI review

This PR is a well-structured, thoroughly tested refactor that fixes the real defect it targets: the base DeterministicIdGenerator was installed as the provider's sole IdGenerator, hijacking trace/span IDs for all instrumentation. The new design scopes deterministic overrides to the plugin's own startSpan calls (via useIds/IdScope + one-shot consumption in generatesRandomTraceIds()), wraps rather than replaces the configured generator (installOn, order()==Integer.MAX_VALUE), late-binds the global provider at invocation start without installing a no-op global, and removes the fabricated replay/continuation links. The test suite is materially strengthened (scoped-delegation, cross-classloader bridge, concurrency, sampler-ordering, independent-root assertions), and the ProviderSource/AUTO_OTLP removals are internally consistent across the module and examples.

I found no confirmed correctness regression in the scoped-ID mechanism, the late-binding flow, or the parent-resolution changes. I raised one design-level correctness/observability risk inline (below).

Residual test risk: The one open concern — the Workflow trace ID's timestamp prefix being the execution start time — is not exercised for genuinely long executions. The conformance workflow's launch phase caps delay_seconds at 86400 (~23h), well within any X-Ray ingestion window, so executions that run for weeks/months (the SDK supports up to 1 year) are not covered end-to-end. Everything else is covered by the added unit/integration tests.

Reviewed commit e6b5f7ee885cfeec46cdccb7242bfdcec48a877b. Workflow run

github-actions · 2026-08-18T23:06:07Z

+ * = "Workflow"}, {@code instrumentationName = "aws-durable-execution-sdk-java"}. A {@code null} passed to any builder
+ * setter falls back to the corresponding default.


Codex AI review

[P1] Preserve the public provider-selection API. This change removes ProviderSource, the provider/OTLP getters and builder methods, and providerSource() from both plugins. Existing clients will fail to compile or encounter linkage errors. Retain and deprecate these APIs while preserving AUTO_OTLP behavior until a major release.

github-actions · 2026-08-18T23:06:08Z

+    private final ThreadLocal<String> extractedTraceId = new ThreadLocal<>();
+    private final ThreadLocal<String> arnDerivedTraceId = new ThreadLocal<>();
    private final ThreadLocal<String> pendingSpanOperationId = new ThreadLocal<>();
    private final ThreadLocal<String> pendingRawSpanId = new ThreadLocal<>();
-    private final AtomicReference<String> durableExecutionArn = new AtomicReference<>(null);
+    private final ThreadLocal<IdOverride> scopedIds = new ThreadLocal<>();
+    private final ThreadLocal<String> durableExecutionArn = new ThreadLocal<>();


Codex AI review

[P2] Keep persistent setter state process-wide. These fields were AtomicReferences, so configuring the public generator once applied to spans created on executor threads. With ThreadLocal, those threads fall back to random trace IDs and an empty execution ARN, breaking cross-thread correlation. Use separate process-wide state for the legacy setters and reserve thread-local state for scoped overrides; add a cross-thread test.

github-actions · 2026-08-18T23:06:09Z

+    }
+
+    String generateTraceIdForExecution(String arn, Instant executionStartTime) {
+        var timestamp = executionStartTime != null ? executionStartTime : Instant.now();


Codex AI review

[P2] Do not use wall-clock time as the missing-timestamp fallback. If executionStartTime is null, later invocations of the same durable execution generate different Workflow trace IDs and split the trace. Derive a stable fallback from the execution ARN, or reject the missing timestamp before emitting telemetry, and cover replay with a null timestamp.

github-actions · 2026-08-18T23:06:11Z

Codex AI review

Found three compatibility and deterministic-ID issues. Static review only; tests were not run per constraints.

Reviewed commit 1efff4053dd477f821759cf5a7cf72c9ef0ca0af. Workflow run

zhongkechen requested a review from a team August 15, 2026 18:30

zhongkechen temporarily deployed to ai-pr-review-runtime August 15, 2026 18:30 — with GitHub Actions Inactive